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MARKED-UP SPECIFICATION 
Express Mail Label No. EV318163135US 

[0001] IMAGE PICKUP SYSTEM 

[0002] CROSS REFERENCE TO RELATED APPLICATION(S) 

[0003] This application claims benefits of Japanese Application No. 2002-242400 

filed in Japan on Aug. 22, 2002, the contents of which are incorporated by this 

reference. 

[0004] BACKGROUND OF THE INVENTION 

[0005] 1. Field of the Invention 

[0006] The present invention relates to an image pickup system and image 

processing program which reduce random noise arising in the image pickup element 
system. 

[0007] 2. Description of the Related Art 

[0008] Generally, noise components are contained in digitized signals obtained 

from image pickup elements and the associated analog circuits and A/D converters. 
Such noise components can be divided into two main categories, i.e., fixed pattern noise 
and random noise. 

[0009] The abovementioned fixed pattern noise, as typified by defective pixels or 

the like, is noise that originates mainly in the image pickup elements. 

[0010] On the other hand, random noise is generated in the image pickup 

elements and analog circuits, and has characteristics that are close to white noise 

characteristics. 

[0011] In regard to the latter random noise, for example, a technique in which 

the amount of noise N is converted into a function by N = ab cD using constant terms a, 
b and c that are given as static terms, and the signal level D converted into a density 
value, the amount of noise N for the signal level D is estimated from this function, and 
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the filtering frequency characteristics are controlled on the basis of the estimated 
amount of noise N, is disclosed in Japanese Patent Application Laid-Open No. 2001- 
157057. Using this technique, an appropriate noise reduction treatment can be 
performed on the signal level. 

[0012] Furthermore, as another example, a technique such that the difference 

value A between a pixel of interest and a nearby pixel is determined, the average pixel 
number n used in the moving average method is controlled by the function n = a/(A + b) 
using the determined difference value A and constant terms a and b that are given as 
static terms, and a moving average is not determined in cases where the determined 
difference value A is equal to or greater than a specified threshold value, is described in 
Japanese Patent Application Laid-Open No. 2002-57900. By using such a technique, it 
is possible to perform a noise reduction treatment without causing any deterioration of 
the original signal such as edges or the like. 

[0013] However, since the amount of noise varies dynamically according to 

factors such as the temperature at the time of shooting, exposure time, gain and the 
like, conversion to a function that matches the amount of noise during shooting cannot 
be handled in the case of a technique using static constant terms such as that described 
in the abovementioned Japanese Patent Application Laid-Open No. 2001-157057, so 
that the precision in estimating the amount of noise is inferior. Furthermore, the 
filtering frequency characteristics are controlled from the amount of noise; however, 
since this filtering performs processing equally without discriminating between flat 
portions and edge portions, the edge portions deteriorate in regions where it is 
estimated on the basis of the signal level that the amount of noise is large. 
Specifically, processing that discriminates between the original signal and noise cannot 
be handled, so that the preservation of the original signal is poor. 
[00 14] Furthermore, in the technique described in Japanese Patent Application 

Laid-Open No. 2002-57900, the determination of whether or not the moving average 
method is performed is accomplished by comparison with a threshold value. However, 
since this threshold value is also given as a static value, variation in the amount of 
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noise according to the signal level cannot be handled, so that the selection of the 
average number of pixels or moving average method cannot be optimally controlled. 
Consequently, noise components remain, and there is a deterioration of the original 
signal and the like. 

[00 15] Furthermore, in cases where there are differences in the conditions during 

shooting or subjects of shooting, e. g., in the case of flat subject of shooting such as skin 
or the like, or subject of shooting that has a texture structure, the subjective evaluation 
may be different even if the amount of noise is the same. However, in the 
abovementioned prior art, such points cannot be handled, so that there is a drawback 
that a subjectively ideal image may not always be obtainable even if noise reduction 
processing is performed. 

[0016] SUMMARY OF THE INVENTION 

[0017] In short, the image pickup system comprises: a noise estimating unit 

which estimates the amount of noise contained in the digitized signal from an image 
pickup element in which a plurality of pixels are arranged, either for each pixel or for 
each specified unit area comprising a plurality of pixels; and a shooting situation 
estimating unit which estimates the shooting situation at the time that an image based 
on the abovementioned signal is shot. The amount of noise estimated by the 
abovementioned noise estimating unit is corrected on the basis of the shot situation 
estimated by the abovementioned shooting situation estimation unit, and the noise in 
the abovementioned signal is reduced on the basis of the corrected amount of noise. 
[0018] Additional features and advantages of the invention will be set forth in 

the description which follows, and in part will be obvious from the description, or may 
be learned by the practice of the invention. The features and advantages of the 
invention may be realized and obtained by means of the instrumentalities and 
combinations particularly pointed out hereinafter. 
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[0019] BRIEF DESCRIPTION OF THE DRAWINGS 

[0020] Fig. 1 is a block diagram which shows the construction of the image 

pickup system in a first embodiment of the present invention; 

[0021] Fig. 2 is a diagram which shows the primary color Bayer type filter 

construction in the color filters of the abovementioned first embodiment; 

[0022] Fig. 3 is a block diagram which shows the construction of the shooting 

situation estimation unit in the abovementioned first embodiment; 

[0023] Fig. 4 is a diagram which is used to illustrate the split pattern used for 

exposure controlling and the parameters used for evaluation in the abovementioned 

first embodiment; 

[0024] Fig. 5 is a block diagram which shows the construction of the noise 

estimating unit in the abovementioned first embodiment; 

[0025] Fig. 6 is a diagram which is used to illustrate the formulization of the 

amount of noise in the abovementioned first embodiment; 

[0026] Fig. 7 is a diagram which is used to illustrate the formulization of the 

amount of noise in the abovementioned first embodiment; 

[0027] Fig. 8A is a diagram which is used to illustrate the parameters used in the 

formulization of the amount of noise in the abovementioned first embodiment; 
[0028] Fig. 8B is a another diagram which is used to illustrate the parameters 

used in the formulization of the amount of noise in the abovementioned first 
embodiment; 

[0029] Fig. 8C is a another diagram which is used to illustrate the parameters 

used in the formulization of the amount of noise in the abovementioned first 
embodiment; 

[0030] Fig. 8D is a still another diagram which is used to illustrate the 

parameters used in the formulization of the amount of noise in the abovementioned 
first embodiment; 

[0031] Fig. 9 is a block diagram which shows the construction of the noise 

reducing unit in the abovementioned first embodiment; 
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[0032] Fig. 10 is a block diagram which shows the construction of the image 

pickup system of a second embodiment of the present invention; 

[0033] Fig. 11A is a block diagram which shows one example of the construction 

of the shooting situation estimating unit in the abovementioned second embodiment; 

[0034] Fig. 11B is a block diagram which shows another example of the 

construction of the shooting situation estimating unit in the abovementioned second 

embodiment; 

[0035] Fig. 12 is a block diagram which shows the construction of the noise 

estimating unit in the abovementioned second embodiment; 

[0036] Fig. 13 is a flow chart which shows one portion of the noise reduction 

processing performed by means of an image processing program in a computer in the 
abovementioned second embodiment; and 

[0037] Fig. 14 is a flow chart which shows another portion of the noise reduction 

processing performed by means of an image processing program in a computer in the 
abovementioned second embodiment. 

[0038] DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0039] Embodiments of the present invention will be described below with 

reference to the attached figures. 

[0040] Figs. 1 through 9 show a first embodiment of the present invention. Fig. 1 

is a block diagram which shows the construction of the image pickup system, Fig. 2 is a 
diagram which shows the primary color Bayer type filter construction in the color 
filters, Fig. 3 is a block diagram which shows the construction of the shooting situation 
estimating unit, Fig. 4 is a diagram which is used to illustrate the split pattern used 
for exposure controlling and the parameters used for evaluation, Fig. 5 is a block 
diagram which shows the construction of the noise estimating unit, Figs. 6 and 7 are 
diagrams which are used to illustrate the formulization of the amount of noise, Figs. 
8A through 8D are diagrams which are used to illustrate the parameters used in the 
formulization of the amount of noise, and Fig. 9 is a block diagram which shows the 
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construction of the noise estimating unit. 

[0041] As is shown in Fig. 1, this image pickup system comprises: a lens system 

1 which is used to form an image of the subject of shooting, an aperture 2 which is 
arranged inside the lens system 1, and which is used to regulate the transmission 
range of the luminous flux in the lens system 1; a low-pass filter 3 which is used to 
eliminate unnecessary high-frequency components from the luminous flux that is 
formed by the abovementioned lens system 1; primary color Bayer type color filters 4, 
for example, which are arranged on the surface of a CCD 5 (described later) in the light 
path of the luminous flux focused by the abovementioned lens system 1; a CCD 5 
constituting a black and white image pickup element which subjects the optical subject 
image that is formed via the color filter 4 to a photoelectric conversion, and outputs an 
electrical video signal; a temperature sensor 6 constituting parameter calculating 
means which is arranged in the vicinity of the abovementioned CCD 5, and which are 
used to measure the temperature of the CCD 5 in real time and output the 
measurement results of a control unit 22 (described later); a CDS (correlated double 
sampling) unit 7 which performs correlated double sampling on the video signal that is 
output from the abovementioned CCD 5; an amplifier 8 which amplifies the signal 
that is output from the CDS unit 7; an A/D converter 9 which converts the analog 
video signal that is amplified by the abovementioned amplifier 8 into a digital signal; 
an image buffer 10 which temporarily stores the digital image data that is output from 
the abovementioned A/D converter 9; an exposure control unit 11 which performs 
exposure controlling relating to the subject on the basis of the image data stored in the 
abovementioned image buffer 10, and controls the abovementioned aperture 2, CCD 5 
and amplifier 8 on the basis of the evaluation results; a pre-WB unit 12 which detects 
a simple white balance on the basis of the video signal stored in the abovementioned 
image buffer 10, and controls the abovementioned amplifier 8 on the basis of the 
results; a focus detection unit 13 which detects the focal point on the basis of the 
image data stored in the abovementioned image buffer 10, and drives an AF motor 14 
(described later) on the basis of the detection results; an the AF motor 14 which is 
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controlled by the abovementioned focus detection unit 13, and which drives a focusing 
lens or the like contained in the abovementioned lens system 1; a color signal 
separating unit 15 constituting separating means which reads out the video signal 
stored in the abovementioned image buffer 10, and separates the color signals; a noise 
estimating unit 17 constituting noise estimating means which estimates noise (as will 
be described in detail later) on the basis of the image data that is output from the 
abovementioned color signal separating unit 15; a shooting oituation conditions 
estimating unit 16 constituting shooting situation conditions estimating means which 
estimates the shooting situation; a correction unit 18 constituting correction means 
which uses the results of the estimation performed by the abovementioned shooting 
situation conditions estimating unit 16 to correct the results of the estimation 
performed by the abovementioned noise estimating unit 17; a noise reducing unit 19 
constituting noise reducing means which reduces the noise of the image data output 
from the abovementioned color signal separating unit 15 using the estimated noise 
corrected by the abovementioned correction unit 18; a signal processing unit 20 which 
processes the image data output from the abovementioned noise reducing unit 19; an 
output unit 21 which outputs the image data from the abovementioned signal 
processing unit 20 in order to record this data on, for example, a memory card or the 
like; an external I/F unit 23 comprising an interface to the power supply switch, 
shutter button, mode switches for switching various shooting modes and the like; and 
a control unit 22 constituting control means comprising a microcomputer or the like 
which is connected bidirectionally with the abovementioned CDS unit 7, amplifier 8, 
A/D converter 9, exposure control unit 11, pre-WB unit 12, focus detection unit 13, color 
signal separating unit 15, shooting situation conditions estimating unit 16, noise 
estimating unit 17, correction unit 18, noise reducing unit 19, signal processing unit 
20, output unit 21 and external I/F unit 23, and which comprehensively controls the 
abovementioned image pickup system containing these parts, and also constituting 
parameter calculating means. 

[0042] Next, the flow of signals in the image pickup system shown in Fig. 1 will 
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be described. 

[0043] The abovementioned image pickup system is constructed such that 

shooting situations conditions such as the ISO sensitivity and the like can be set via 
the external I/F unit 23; after these settings have been accomplished, the pre-image- 
pickup mode is entered by pressing the shutter button, which is a two-stage push 
button switch. 

[0044] The video signal that is imaged by the CCD 5 and output via the 

abovementioned lens system 1, aperture 2, low-pass filter 3 and color filters 4 A is 
subjected to universally known correlated double sampling in the CDS unit 7 and read 
out as an analog signal. 

[0045] This analog signal is amplified by a specified amount by the amplifier 8, 

converted into a digital signal by the A/D converter 9, and transmitted to the image 
buffer 10. 

[0046] The video signal inside the image buffer 10 is then respectively 

transmitted to the exposure control unit 11, the pre-WB unit 12 and the focus detection 
unit 13. 

[0047] The exposure control unit 11 determines the brightness levels in the 

image, splits the image into a plurality of regions with the set ISO sensitivity, hand 
movement limit shutter speed and the like taken into account, calculates the 
appropriate exposure value by combining the brightness levels of the respective 
regions, and controls the stop value of the aperture 2, the electronic shutter speed of 
the CCD 5, the amplification factor of the amplifier 8 and the like such that this an 
appropriate exposure value is obtained. 

[0048] Furthermore, the pre-WB unit 12 calculates a simple white balance by 

multiplying a signal of a specified level in the video signal by each color signal, 
transmits the result to the amplifier 8, and achieves a white balance by multiplying 
this value by a gain that differs for each color signal. 

[0049] Then, the focus detection unit 13 detects the edge intensity in the image, 

and obtains a focused image by controlling the AF motor 14 such that this edge 
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intensity is maximized. 

[0050] When preparations for the real shooting have been completed by 

performing this pre-image-pickup mode, and it is then detected via the external I/F 
unit 23 that the shutter button has been fully pressed, real shooting is performed. 
[0051] This real shooting is performed on the basis of the exposure conditions 

determined by the exposure control unit 11, the white balance coefficients determined 
by the pre-WB unit 12, and the focusing conditions determined by the focus detection 
unit 13, and these shooting situations are transmitted to the control unit 22. 
[0052] When real shooting is thus performed, the video signal is transmitted to 

the image buffer 10 and stored in the same manner as in pre-image-pickup. 
[0053] The video signal inside the image buffer 10 is transmitted to the color 

signal separating unit 15, and is separated into the respective colors of the color filters. 
[0054] As is described above, the filter construction of the abovementioned color 

filters 4 arranged on the front surface of the CCD 5 is as follows: for example, these 
are primary color Bayer type niters of the type shown in Fig. 2, i.e., filters in which the 
basic unit is 2 x 2 pixels, with green (Gl and G2) filters arranged in diagonal positions, 
and red (R) and blue (B) filters arranged in the remaining diagonal positions. 
Furthermore, the green filters Gl and G2 are filter filters with having the same optical 
characteristics; here, however, these filters are distinguished as Gl and G2 for 
convenience of processing. 

[0055] The color signal separating unit 15 separates the video signal inside the 

image buffer 10 in accordance with these four types of color filters R, Gl, G2 and B, 
and this separating operation is performed in synchronization with the processing of 
the noise estimating unit 17 and the processing of the noise reducing unit 19 under the 
control of the control unit 22. 

[0056] Meanwhile, the control unit 22 transmits exposure information and focus 

information at the time of image pickup from the exposure control unit 11, pre-WB unit 
12 and focus detection unit 13 to the shooting situation conditions estimating unit 16. 
[0057] On the basis of this transmitted information, the shooting situation 
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conditions estimating unit 16 estimates the shooting situation conditions for the entire 
signal, e.g., typically including conditions such as scenery shooting, portrait shooting, 
close-up shooting, night view shooting or the like, and transmits the conditions of this 
shooting situation to the correction unit 18. Such estimation processing of the shooting 
situation by the shooting situation conditions estimating unit 16 is performed once for 
each shooting operation. 

[0058] Next, the noise estimating unit 17 reads in the respective color signals 

from the color signal separating unit 15 under the control of the control unit 22; 
furthermore, shooting situations conditions such as the exposure conditions determined 
by the exposure control unit 11, the ISO sensitivity set by the external I/F unit 23 and 
the like are also transmitted to the noise estimating unit 17 via the control unit 22. 
[0059] On the basis of the abovementioned information and respective color 

signals, the noise estimating unit 17 calculates the amount of noise for each specified 
size, e. g., for each pixel (in pixel units) in the present embodiment, and transmits the 
calculated amount of noise to the correction unit 18. 

[0060] On the basis of the shooting situation conditions output from the shooting 

situation conditions estimating unit 16, the correction unit 18 corrects the amount of 
noise output from the noise estimating unit 17; then, the correction unit 18 transmits 
the corrected amount of noise to the noise reducing unit 19. 

[0061] In this case, the processing in the abovementioned noise estimating unit 

17 and the processing in the correction unit 18 are performed in synchronization with 
the processing of the noise reducing unit 19 under the control of the control unit 22. 
[0062] On the basis of the amount of noise corrected by the correction unit 18, the 

noise reducing unit 19 performs noise reduction processing on the respective color 
signals from the color signal separating unit 15, and transmits the processed video 
signal to the signal processing unit 20. 

[0063] Under the control of the control unit 22, the signal processing unit 20 

performs universally known emphasis processing, compression processing and the like 
on the video signal that has been subjected to noise reduction, and transmits the 
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processed video signal to the output unit 21. 

[0064] The output unit 21 records and stores the received video signal on a 

memory card or the like. 

[0065] Next, one example of the construction of the shooting situation conditions 

estimating unit 16 will be described with reference to Fig. 3. 

[0066] The shooting situation conditions estimating unit 16 comprises: a 

focusing position estimating unit 31 constituting focusing position estimating means 
that acquires the AF information set by the abovementioned focus detection unit 13 
from the control unit 22, and classify classifies this information into, for example, 5 m 
to infinity (scenery shooting), 1 m to 5 m (portrait shooting), 1 m and less (close-up 
shooting) or the like; a shooting-subject distribution estimating unit 32 constituting 
shooting-subject distribution estimating means that acquires the split photometrical 
results of the abovementioned exposure control unit 11 from the control unit 22 and 
calculates evaluation parameters SI through S3 (described later); a night view 
estimating unit 33 which acquires AE information from the abovementioned exposure 
control unit 11 via the control unit 22, and estimates that the shooting situation is 
night view shooting in cases where the exposure time is longer than a specified shutter 
speed, and the average brightness level for the overall signal is equal to or less than a 
specified threshold value; and an overall estimating unit 34 constituting overall 
estimating means that determines the gain used to perform a correction for the amount 
of noise on the basis of the results of the classification performed by the 
abovementioned focusing position estimating unit 31, the results of the estimation 
performed by the abovementioned shooting-subject distribution estimating unit 32 and 
the results of the estimation performed by the abovementioned night view estimating 
unit 33, and transfers the gain thus determined to the correction unit 18. 
[0067] Furthermore, the abovementioned focusing position estimating unit 31, 

shooting-subject distribution estimating unit 32 and night view estimating unit 33 are 
connected bidirectionally with the control unit 22. 

[0068] For example, as is shown in Fig. 4, the abovementioned exposure control 
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unit 11 splits the signal from the CCD 5 into 13 regions, and performs universally 
known split photometry. 

[0069] In the example shown in the figures, the image pickup region of the CCD 

5 is classified into a centermost portion, an inner peripheral portion that surrounds 
this centermost portion, and an outer peripheral portion that surrounds this inner 
peripheral portion. Furthermore, these portions are further divided into regions as 
described below. 

[0070] Specifically, the centermost portion is divided into a middle region (with 

an average brightness level indicated by al), a region adjacent to this middle region on 
the left (with an average brightness level indicated by a2), and a region adjacent to this 
middle region on the right (with an average brightness level indicated by a3). 
[0071] Furthermore, the inner peripheral portion is divided into regions (with 

average brightness levels respectively indicated by a4 and a5) above and below the 
region with an average brightness level of al, regions (with average brightness levels 
respectively indicated by a6 and a7) located to the left and right of the region with an 
average brightness level of a4, and regions (with average brightness levels respectively 
indicated by a8 and a9) located to the left and right of the region with an average 
brightness level of a5. 

[0072] Furthermore, the outer peripheral portion is divided into an upper left 

region (with an average brightness level indicated by alO), an upper right region (with 
an average brightness level indicated by all), a lower left region (with an average 
brightness level indicated by al2), and a lower right region (with an average 
brightness level indicated by al3). 

[0073] The average brightness levels of the respective regions produced by such a 

division are transmitted to the shooting subject distribution estimating unit 32 from 
the exposure control unit 11. 

[0074] In the split photometry using such regions, the abovementioned shooting- 

subject distribution estimating unit 32 calculates the respective evaluation parameters 
shown below as shown in the following equation 1, equation 2 and equation 3, and 
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transmits the results of these calculations to the overall estimating unit 34. 

51 = |a2-a3| ... (1) 

52 = max(|a4-a6|, |a4-a7|) ... (2) 

53 = max(alO, all)-Av ... (3) 
where 

Av = (Zai) / 13 

Here, max() is a function which gives the maximum value of the number in 
parentheses, Z indicates the sum for all i (i.e., i = 1 to 13), and Av indicates the average 
brightness level for all of the photometrical regions (the average brightness level for 
the overall signal). 

[0075] Thus, the evaluation parameter SI indicates the left-right brightness 

difference of the centermost portion (central portion), the evaluation parameter S2 
indicates the larger of the brightness differences between the upper side center and 
upper side left or between the upper side center and upper side right of the inner 
peripheral portion (central emphasis), and the evaluation parameter S3 indicates the 
difference between the average brightness of the overall signal and whichever is larger 
of the upper side left and the upper side right of the outer peripheral portion (overall 
signal). 

[0076] As is described above, the abovementioned overall estimating unit 34 
determines the gain that is used to correct the amount of noise in accordance with the 
respective outputs of the abovementioned focusing position estimating unit 31, the 
shooting-subject estimating unit 32 and the abovementioned night view estimating 
unit 33. Here, in cases where the estimation results from the night view estimating 
unit 33 indicate night view shooting, "strong", e. g., a value of 1.5 to 2.0, is designated 
as the gain; this gain is immediately sent to the correction unit 18, and the processing 
is ended. 

[0077] On the other hand, in cases where it is estimated that the shooting is not 

night view shooting, the overall estimating unit 34 estimates the shooting situation 
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and determines the gain as shown in Table 1 using the classification results from the 
focusing position estimating unit 31, and the evaluation parameters SI through S3 
from the shooting-subject distribution estimating unit 32. 



Table 1 



AF 

INFORMATION 


AF 

INFORMATION 


SHOOTING SCENE 


GAIN 


5 m ~ co 


S3 > Thl 


SCENERY WITH SKY IN UPPER 
PART 


STRONG 


S3 < Thl 


SCENERY WITH NO (OR LITTLE) 
SKY IN UPPER PART 


MEDIUM 


1 m ~ 5 m 


S2 > Th2 


PORTRAIT OF ONE PERSON 


MEDIUM 


S2 < Th2 


PLURALITY OF PORTRAITS 


STRONG 


1 m OR LESS 


SI > Th3 


CLOSE-UP OF PLURALITY OF 
OBJECTS 


WEAK 


SI < Th3 


CLOSE-UP OF SINGLE OBJECT 


MEDIUM 



[0078] As is shown in Table 1, when the AF information is 5 m to infinity, the 

shooting subject is considered to be scenery, and the abovementioned evaluation 
parameter S3 is further compared with a specified value Thl. In this case, if the 
evaluation parameter S3 is larger than the specified value Thl, at least one of the 
values alO or all shows a brightness that is higher by at least a certain amount than 
the average brightness of the overall signal; accordingly, it is estimated that the 
shooting involves scenery with sky in the upper part. Since the sky is flat and this is a 
region where noise components are subjectively a problem, a "strong" gain (e. g., 1.5 to 
2.0) is designated as the gain used for correction. On the other hand, in cases where 
the evaluation parameter S3 is smaller than the specified value Thl, it is conversely 
estimated that the shooting situation is scenery with no sky (or a little sky) in the 
upper part. In this case, it appears that the main shooting subject is an object with a 
texture textured structure such as plants, buildings or the like; accordingly, a 
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"medium" gain (e. g., 1.0 to 1.5) is designated as the gain used for correction. 
[0079] Next, in cases where the AF information is 1 m to 5 m, it is assumed that 

the shooting is for portrait, and the abovementioned evaluation parameter S2 is 
further compared with a specified value Th2. In this case, if the evaluation parameter 
S2 is larger than the specified value Th2, there is a brightness difference between the 
upper side center a4 of the inner peripheral portion and either the upper side left or 
right a6 or a7 of the inner peripheral portion; accordingly, it is estimated that the 
shooting situation is in portrait shooting of a single person. In the case of portrait 
shooting of a single person, the area of the face, i.e., the area which is flat and in which 
noise tends to be conspicuous, is relatively large. Accordingly, while it is desirable on 
the one hand to strengthen the gain used for correction, hair which has a fine structure 
is also present; accordingly, if this hair is solid, this will be evaluated as a deterioration 
in the image quality. Consequently, a "medium" gain is designated as the gain used 
for correction. On the other hand, in cases where the evaluation parameter S2 is 
smaller than the specified value Th2, it is estimated that the shooting situation is for 
portrait of a plurality of persons. In the case of portrait shooting of a plurality of 
persons, the area of the faces is relatively small, and the fine structure of the hair is 
difficult to distinguish. Accordingly, a "strong" gain is designated as the gain used for 
correction. 

[0080] Furthermore, in cases where the AF information is 1 m or less, it is judged 

that the shooting is for close-up, and the abovementioned evaluation parameter SI is 
further compared with a specified value Th3. In this case, if the evaluation parameter 
SI is larger than the specified value Th3, there is a brightness difference between the 
left and right of the centermost portion, and it is estimated that the shooting situation 
is for close-up of a plurality of objects. In this case, it would appear that there is fine 
structure in the principle objects of shooting; accordingly, a "weak" gain (e. g., 0.5 to 
1.0) is designated as the gain used for correction. On the other hand, in cases where 
the evaluation parameter SI is smaller than the specified value Th3, it is estimated 
that the shooting situation is for close-up of a single object. In this case, it is difficult to 
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judge the presence or absence of fine structure; accordingly, all-purpose characteristics 
are taken into account, and a "medium" gain is designated as the gain used for 
correction. 

[0081] The gain used for correction that has thus been set by the overall 

estimating unit 34 is transmitted to the correction unit 18 as described above. 
[0082] Next, one example of the construction of the noise estimating unit 17 will 

be described with reference to Fig. 5. 

[0083] The noise estimating unit 17 comprises: a local region extraction unit 41 

which extracts local regions of a specified size in specified positions in each color signal 
that is output from the abovementioned color signal separating unit 15; a buffer 42 
which stores the color signals of the local regions extracted by the local region 
extraction unit 41; a gain calculating unit 44 constituting parameter calculating 
means that calculates the amount of amplification of the abovementioned amplifier 8 
on the basis of information relating to the exposure conditions and information relating 
to the white balance coefficients transmitted from the abovementioned control unit 22; 
a standard value assigning unit 45 constituting assigning means that assigns 
standard values in cases where any of the parameters are omitted; an average 
calculating unit 43 constituting parameter calculating means that reads out the 
signals of the local regions stored in the abovementioned buffer 42, calculates average 
values and transmits these values to a coefficient calculating unit 46 as signal value 
levels; a pattern parameter ROM 47 constituting coefficient calculating means that 
stores parameters relating to functions (described later) that are used to estimate the 
amount of noise; a coefficient calculating unit 46 serving as coefficient calculating 
means as well as noise amount calculating means that calculates coefficients involved 
in specified equations that are used to estimate the amount of noise of pixels of interest 
on the basis of the parameters that are read out from the abovementioned parameter 
ROM 47, information relating to the temperature of the image pickup element that is 
output from the temperature sensor 6 or standard value assigning unit 45 via the 
abovementioned control unit 22, signal value levels that are output from the 
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abovementioned average calculating unit 43 or the abovementioned standard value 
assigning unit 45, the amount of amplification that is output from the abovementioned 
gain calculating unit 44 or the abovementioned standard value assigning unit 45, and 
information relating to the shutter speed that is output from the abovementioned 
control unit 22 or the abovementioned standard value assigning unit 45; and a 
function calculating unit 48 serving as function calculating means as well as noise 
amount calculating means that calculates the amount of noise using functions that are 
formulized (as will be described later) using the coefficients that are output from the 
abovementioned coefficient calculating unit 46 and transfers the amount of noise thus 
calculated to the correction unit 18. 

[0084] In the present embodiment, since the processing of the noise reducing unit 

19 (as will be described later) is separated into the horizontal direction and vertical 
direction, the abovementioned local region extraction unit 41 is devised such that this 
unit performs extraction while successively scanning the entire image in, for example, 
4x1 size units in the case of horizontal direction processing, and, for example, 1x4 
size units in the case of vertical direction processing. The processing performed by this 
local region extraction unit 41 is performed in synchronization with the processing of 
the noise reducing unit 19. 

[0085] Furthermore, the abovementioned control unit 22 is connected 

bidirectionally with the abovementioned local region extraction unit 41, average 
calculating unit 43, gain calculating unit 44, standard value assigning unit 45, 
coefficient calculating unit 46, and function calculating unit 48, and controls these 
units. 

[0086] Next, the formulization of the amount of noise that is used when the 

coefficient calculating unit 46 estimates the amount of noise for pixels of interest will 
be described with reference to Fig. 7. 

[0087] The function of the amount of noise N with respect to the signal value 
level L is formulized as indicated in equation 4 below. 
N = AL B + C (4) 
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Here, A, B and C are constant terms, and the constant terms are added to a function 
that forms a power of the signal value level L. 

[0088] When the outline of this function is plotted in a cases case where (for 

example) A>0, 0<B<1 and C > 0, the shape shown in Fig. 6 is obtained. 
[0089] However, the amount of noise N does not depend on the signal value level 

L alone, but also varies according to the temperature of the CCD 5 which is the image 
pickup element and the gain of the amplifier 8. Accordingly, Fig. 7 shows an example 
in which these factors are also taken into account. 

[0090] Specifically, as is shown in equation 5, a(T, G), b(T, G) and c(T, G) which 
use the temperature T and gain G as parameters, are introduced instead of A, B and C, 
which were constant terms on the abovementioned equation 4. 
N = a(T, G)L b(T ' « + C (T, G) (5) 

[0091] Fig. 7 shows the curves indicated by this equation 5 plotted for a plurality 

of gains G (1, 2 and 4 times in the example shown in the figures) at a plurality of 
temperatures T (temperatures Tl through T3 in the example shown in the figures). 
[0092] In Fig. 7, the independent variable is shown as the signal value level L, 

and the dependent variable is shown as the amount of noise N. The temperature T 
which constitutes a parameter is plotted on a coordinate axis in a direction that is 
perpendicular to these variables. Accordingly, the amount of noise N according to the 
signal value level L is respectively read within the plane expressed by T = Tl, within 
the plane expressed by T = T2, and within the plane expressed by T = T3. In this case, 
furthermore, the curve shape of the gain G, which is a parameter, is expressed by 
drawing a plurality of curves within the respective planes. 

[0093] The individual curves indicated by the respective parameters show 

configurations that are more or less similar to the curves produced by equation 4 
shown in Fig. 6; however, the respective coefficients a, b and c naturally differ 
according to the respective values of the temperature T and gain G. 
[0094] Fig. 8A shows an outline of the characteristics of the abovementioned 

function a(T, G), Fig. 8B shows an outline of the characteristics of the abovementioned 
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function b(T, G), and Fig. 8C shows an outline of the characteristics of the 
abovementioned function c(T, G). 

[0095] Since these respective functions are two-variable functions with the 

temperature T and gain G as independent variables, Figs. 8A through 8C are plotted 
as three-dimensional coordinates, and show curved surfaces in these plotted spaces. 
Here, however, instead of showing concrete curved shapes, the conditions of major 
changes in the characteristics are shown using curved lines. 

[0096] The respective constant terms A, B and C are output by inputting the 

temperature T and gain G into such functions a, b and c as parameters. Furthermore, 
the concrete shapes of these functions can easily be acquired by measuring the 
characteristics of the image pickup element system including the CCD 5 and amplifier 
8 beforehand. 

[0097] Random noise shows a tendency to increase as the exposure time becomes 

longer. As a result, if the combination of shutter speed and stop value differs, there 
may be a difference in the amount of noise that is generated, even if the amount of 
exposure is the same. Accordingly, an example in which correction is performed with 
such differences also being taken into account will be described with reference to Fig. 
8D. 

[0098] Here, a correction coefficient d(S) which gives the constant term D is 

introduced with the shutter speed S as a parameter, and a correction based on the 
formulization shown in equation 6 is performed by means that multiplies this 
correction coefficient by equation 5. 

N = {a(T, G)Lb(T, G) + c(T, G)}d(S) = (ALB + C)D .... (6) 

[0099] The function shape of this correction coefficient d(S) is obtained by 

measuring the characteristics of the image pickup element system beforehand; for 
example, this function shows has the shape indicated in Fig. 8D. Fig. 8D shows the 
conditions of the increment D in the amount of noise relative to the shutter speed S. 
[00100] As is shown in the Fig. 8D, the increment D in the amount of noise has 
the property of increasing abruptly when the shutter speed S is smaller than a certain 
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threshold value Sth (i.e., when the exposure time is long). Accordingly, the system is 
divided into two procedures depending on whether or not the shutter speed S is equal 
to or greater than this threshold value Sth. In the case of a long exposure, the function 
d(S) is used; however, in the case of a short exposure, the system may be simplified by 
using fixed coefficients. 

[00101] The four functions a(T, G), b(T, G), c(T, G) and d(S) described above are 
recorded in the abovementioned buffer ROM 47. Furthermore, the correction for the 
shutter speed need not necessarily be prepared as a function; it would also be possible 
to prepare this correction by some other means, e. g., as a table or the like. 
[00102] The coefficient calculating unit 46 calculates the respective fixed terms A, 
B, C and D using the four functions recorded in the parameter ROM 47, with the 
temperature T, gain G and shutter speed S acquired dynamically (or acquired from the 
standard value assigning unit 45) as input parameters, and transmits these constant 
terms to the function calculating unit 48. 

[00103] The function calculating unit 48 determines the function shape used to 
calculate the amount of noise N by applying the respective constant terms A, B, C and 
D calculated by the abovementioned coefficient calculating unit 46 to the 
abovementioned equation 6, and calculates the amount of noise N by means of the 
signal value level L output from the abovementioned average calculating unit 43 via 
the abovementioned coefficient calculating unit 46. 

[00 104] In this case, the respective parameters such as the temperature T, gain G, 
shutter speed S and the like need not always be determined for each shooting 
operation. It is also possible to construct the system such that arbitrary parameters 
are stored in the standard value assigning unit 45, and calculation processing is 
omitted. As a result, it is possible to achieve an increase in the processing speed, a 
saving of power and the like. 

[00105] When the abovementioned correction unit 18 receives the amount of noise 
from the noise estimating unit 17 that has been calculated as described above, the 
correction unit 18 multiplies this amount of noise by the correction gain that has been 
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transmitted from the abovementioned shooting situation estimation unit 16 under the 
control of the control unit 22, and transmits the results to the noise reducing unit 19. 
[00106] Next, one example of the construction of the noise reducing unit 19 will be 
described with reference to Fig. 9. 

[00107] The noise reducing unit 19 comprises: a horizontal line extraction unit 5 1 
which successively extracts video signals in horizontal line units for each video signal 
that is output from the abovementioned color signal separating unit 15; a first 
smoothing unit 52 constituting smoothing means which scans in pixel units the 
horizontal line video signals extracted by the abovementioned horizontal line 
extraction unit 51, and performs universally known hysteresis smoothing with the 
threshold value from a threshold value setting unit 56 ( as will be described later) as 
the amount of noise; a buffer 53 which stores the video signals of one screen for all 
colors by successively storing the horizontal lines smoothed by the abovementioned 
first smoothing unit 52; a vertical line extraction unit 54 which successively extracts a 
video signal for each color signal in vertical line units from the abovementioned buffer 
53 after video signals for one screen have been accumulated for all colors in this buffer 
53; a second smoothing unit 55 constituting smooth smoothing means which 
successively scans in pixel units the vertical line video signals extracted by the 
abovementioned vertical line extraction unit 54, performs universally known hysteresis 
smoothing with the threshold value from a threshold value setting unit 56 (to be 
described later) as the amount of noise, and successively outputs the smoothed signals 
to the abovementioned signal processing unit 20; and a the threshold value setting 
unit 56 constituting threshold value setting means which acquires the amount of noise 
corrected by the abovementioned correction unit 18 in pixel units in accordance with 
the horizontal lines extracted by the abovementioned horizontal line extraction unit 51 
or the vertical lines extracted by the abovementioned vertical line extraction unit 54, 
sets the amplitude value of the noise as a threshold value, very small amplitude value, 
and outputs this threshold value to the abovementioned first smoothing unit 52 or the 
abovementioned second smoothing unit 55. 
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[00108] Here, the hysteresis smoothing performed by the abovementioned first 
and second smoothing units 52 and 55 is performed in synchronization with the 
operation of the correction unit 18 and the operation of the threshold value setting unit 
56 under the control of the control unit 22. 

[00109] Furthermore, the abovementioned control unit 22 is connected 
bidirectionally with the abovementioned horizontal line extraction unit 51, vertical line 
extraction unit 54 and threshold value setting unit 56, and controls these units. 
[00 110] Furthermore, in the above description, the amount of noise is estimated in 
pixel units. However, the present invention is not limited to this; it would also be 
possible to devise the system such that the amount of noise is estimated for each 
arbitrary specified unit area such as 2 x 2 pixels, 4x4 pixels or the like. In this case, 
the precision of noise estimation drops; on the other hand, however, this offers the 
advantage of allowing high-speed processing. 

[00111] Furthermore, in the above description, a single CCD in which the color 
filters 4 were primary color Bayer type filters was described as an example. However, 
the present invention is not limited to this; for example, the present invention can also 
be similarly applied to a single CCD in which the color filters 4 are complementary 
color filters. Furthermore, the present invention can also be similarly applied in the 
case of a two (2) CCD CCDs or three (3} CCD CCDs . 

[00112] Furthermore, in the above description, focus information and exposure 
information are used to estimate the shooting situation. However, the present 
invention is not limited to this; it would also be possible to estimate the shooting 
situation using at least one type of information selected from among zoom position 
information, eye sensing information, strobe light emission information and the like, or 
to estimate the shooting situation more precisely by appropriately combining such 
information. 

[00113] In this first embodiment, the amount of noise is estimated for each region 
of the image and for each color signal; accordingly, appropriate noise reduction can be 
accomplished in light areas and dark areas, so that a high-quality image can be 
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obtained. 

[00114] Furthermore, various types of parameters such as the signal value level 
relating to the amount of noise, the temperature of the image pickup element during 
shooting, the shutter speed, the gain and the like are determined dynamically for each 
shooting operation, and the amount of noise is calculated on the basis of these 
parameters; accordingly, the amount of noise can be calculated with high precision. 
In this case, the precision can be increased even further by estimating the amount of 
noise in pixel units. 

[00115] Furthermore, since the shooting situation is determined and the 
estimated amount of noise is corrected, images with subjectively desirable high quality 
can be obtained. In this case, the shooting situation of the overall image is determined 
by synthesizing various types of information during shooting; accordingly, a low cost 
and high-speed processing can be realized. 

[00116] By using focus information, exposure information and the like to 
determine the shooting situation, it is possible to estimate whether or not the shooting 
is for a night scene, and to estimate the category, i.e., close-up, portrait, scenery or the 
like, into which the shooting is to be classified. 

[00117] Furthermore, since signals from an image pickup element that has color 
filters are separated into color signals for each color filter, it is possible to accomplish 
noise reduction in diverse image pickup systems, e. g., primary color systems or 
complementary color systems, and single CCD, two CCD or three CCD systems. 
[00118] Furthermore, since the amount of noise is set as a threshold value, and 
signals below this threshold value are eliminated as noise, signals above this threshold 
value are preserved as original signals, so that a high-quality image in which only the 
noise components are reduced can be obtained. 

[001 19] Furthermore, since standard values are set for parameters that could not 
be obtained at the time of shooting, and since the coefficients used to calculate the 
amount of noise are determined using these standard values together with the 
parameters that are obtained, and the amount of noise is calculated from these 
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coefficients, the amount of noise can be estimated and a stable noise reducing effect can 
be obtained even in cases where necessary parameters cannot be obtained at the time 
of shooting. In this case, since functions are used to calculate the amount of noise, the 
amount of memory that is required can be reduced, so that the cost can be reduced. In 
addition, an image pickup system in which the cost is reduced and power is saved can 
be constructed by deliberately omitting some of the parameter calculations. 
[00120] Figs. 10 through 14 show a second embodiment of the present invention. 
Fig. 10 is a block diagram which shows the construction of the image pickup system, 
Figs. 11A and 11B are block diagrams which show examples of the construction of the 
shooting situation conditions estimation unit, Fig. 12 is a block diagram which shows 
the construction of the noise estimating unit, Fig. 13 is a flow chart which shows a 
another portion of the noise reduction processing that is performed by the image 
processing program in a computer, and Fig. 14 is a flow chart which shows a portion of 
the noise reduction processing that is performed by the image processing program in a 
computer. 

[00121] In this second embodiment, parts that are the same as in the 
abovementioned first embodiment are labeled with the same symbols, and a 
description of such parts is omitted. For the most part, only points that are different 
will be described. 

[00122] As is shown in Fig. 10, in addition to the construction of the 
abovementioned first embodiment, the image pickup system of this second embodiment 
comprises: a down sampling unit 61 comprising down sampling means which thins the 
video signals read out form from the abovementioned image buffer 10 at a specified 
interval; an interpolating unit 62 which performs universally known linear 
interpolation processing on the video signals down sampled by the abovementioned 
down sampling unit 61, thus producing an RGB three CCD image, and outputs the 
result to the abovementioned shooting situation conditions estimation unit 16; and a 
buffer 63 which temporarily stores the information for the labeled image regions 
estimated by the abovementioned shooting situation conditions estimation unit 16, and 
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outputs this information to the abovementioned correction unit 18. The 
abovementioned interpolating unit 62 is connected bidirectionally with the 
abovementioned control unit 22, and is controlled by the control unit 22. 
[00123] The flow of signals in the image pickup system shown in Fig. 10 is 
basically the same as in the abovementioned first embodiment; only portions that are 
different will be described. 

[00124] When it is detected via the external I/F unit 23 that the shutter button 
has been fully pressed, real shooting is performed as described above, and a video 
signal is transmitted to the image buffer 10. 

[00125] The down sampling unit 61 reads out the video signal inside the image 
buffer 10, thins this signal at a specified interval, and transmits the signal to the 
interpolating unit 62. Since Bayer type color filters are premised to be the color filters 
4 in the present embodiment, the down sampling processing performed by the down 
sampling unit 61 is performed with 2x2 pixels as the basic unit. In concrete terms, for 
example, down sampling which reads in only the upper left 2x2 pixels for 16 x 16 
pixel units is performed. As a result of such down sampling processing being 
performed, the video signal is reduced to a size of (1/8) x (1/8), i.e., a data size of 1/64. 
[00126] The interpolating unit 62 produces an RGB three CCD image by 
performing universally known linear interpolation processing on the video signal that 
has been sampled down by the abovementioned down sampling unit 61, and transmits 
the video signal of the three CCD image thus produced to the shooting situation 
estimation unit 16. 

[00127] The shooting situation conditions estimation unit 16 calculates 
information relating to skin color, dark portions, high-frequency regions and the like 
from the video image converted into a three CCD image that is transmitted from the 
interpolating unit 62, divides the shooting image into a plurality of regions on the basis 
of the calculated information, and labels the information for the respective regions. 
[00128] The buffer 63 stores the information transmitted from the shooting 
situation estimation unit 16. 
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[00129] Under the control of the control unit 22, the noise estimating unit 17 
calculates the amount of noise for the respective color signals received from the color 
signal separating unit 15 for each specified size, e. g., in pixel units in the present 
embodiment, and transmits the calculated amount of noise to the correction unit 18. 
[00130] On the basis of the label information read out from the abovementioned 
buffer 63, the correction unit 18 corrects the amount of noise that is output from the 
noise estimating unit 17, and transmits the corrected amount of noise to the noise 
reducing unit 19. In this case, correction unit 18 subjects the label information of the 
buffer 63 to expansion processing in accordance with the proportion of down sampling 
performed by the down sampling unit 61, and performs processing corresponding to the 
amount of noso noise in pixel units from the noise estimating unit 17. 
[00131] Under the control of the control unit 22, the processing in the 
abovementioned noise estimating unit 17 and the processing in the correction unit 18 
are performed in synchronization with the processing of the noise reducing unit 19. 
[00 132] The subsequent processing performed in the noise reducing unit 19, signal 
processing unit 20 and output unit 21 is the same as in the abovementioned first 
embodiment. 

[00133] Next, one example of the construction of the shooting situation conditions 
estimation unit 16 will be described with reference to Fig. 11 A. 

[00134] The shooting situation conditions estimation unit 16 comprises: a skin 
color detection unit 71 constituting specific color detection means which is image 
characteristic detection means that reads out RGB three CCD images from the 
abovementioned interpolating unit 62, calculates color difference signals Cb and Cr, 
extracts skin color regions by specified threshold value processing, and applies labels; 
a dark part (i.e., area) area detection unit 72 constituting specific brightness detection 
means which is image characteristic detection means that reads out RGB three CCD 
images from the abovementioned interpolating unit 62, calculates a brightness signal 
Y, extracts dark regions which are smaller than a specified threshold value, and 
applies labels; and a region estimating unit 73 constituting region estimating means 
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that transmits region estimation results, which are derived on the basis of information 
from the abovementioned skin color detection unit 71 and information from the dark 
area detection unit 72 and indicate whether or not certain regions are skin color 
regions and whether or not these regions are dark regions, to the abovementioned 
buffer 63. 

[00135] Furthermore, the abovementioned control unit 22 is connected 
bidirectionally with the abovementioned skin color detection unit 71 and dark area 
detection unit 72, and controls these units. 

[00136] When an RGB three CCD image is read out from the interpolating unit 
62, the abovementioned skin color detection unit 71 converts this image into signals of 
specific color spaces, e. g., color difference signals Cb and Cr in Y, Cb and Cr spaces, as 
shown in equation 7 and equation 8 below. 

Cb = -0.16874R - 0.33126G + 0.50000B... (7) 

Cr = 0.50000R - 0.41869G - 0.08131B (8) 

[00137] Then, the skin color detection unit 71 extracts only skin color regions by 
comparing these two color difference signals Cb and Cr with a specified threshold 
value. 

[00138] Furthermore, using these results, the skin color detection unit 71 labels 
the down sampled three CCD image in pixel units, and transmits this image to the 
region estimating unit 73. In regard to the concrete labels that are applied in this case, 
for example, 1 is applied to skin color regions, and 0 is applied to other regions. 
[00139] Next, when an RGB three CCD image is read out from the interpolating 
unit 62, the abovementioned dark area detection unit 72 converts this into a brightness 
signal as shown in the following equation 9. 

Y = 0.29900R + 0.58700G + 0.11400B ... (9) 

[00140] Furthermore, the dark area detection unit 72 compares this brightness 
signal Y with a specified threshold value, and extracts regions that are smaller than 
this threshold value as dark regions. 
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[00141] Furthermore, using these results, the dark area detection unit 72 labels 
the down sampled three CCD image in pixel units, and transmits this image to the 
region estimating unit 73. fe With regard to the concrete labels that are applied in 
this case, for example, 2 is applied to dark regions, and 0 is applied to other regions. 
[00142] On the basis of information from the skin color detection unit 71 and 
information from the dark area detection unit 72, the region estimating unit 73 outputs 
skin color regions as 1, dark regions as 2, regions that are both skin color regions and 
dark regions as 3, and other regions as 0, to the buffer 63. 

[00 143] The correction unit 18 reads out this label information from the buffer 63, 
and adjusts the gain used for correction in accordance with the values of the labels. 
For example, in the case of the label 3 (skin color region and dark region), the gain is 
set as "strong" (e. g., 1.5 to 2.0), in the case of the label 1 (skin color region) or label 2 
(dark region), the gain is set as "medium" (e. g., 1.0 to 1.5), and in the case of the label 
0 (other), the gain is set as "weak" (e. g., 0.5 to 1.0). 

[00144] Furthermore, in the example shown in Fig. 11A, color information and 
brightness information are used to estimate the shooting situation of the respective 
regions in the image. However, the present invention is not limited to the use of such 
information; for example, it would also be possible to use frequency information as 
shown in Fig. 11B. 

[00 145] Another example of the construction of the shooting situation conditions 
estimation unit 16 will be described with reference to Fig. 11B. 
[00146] The shooting situation conditions estimation unit 16 comprises: a high- 
frequency detection unit 75 constituting frequency detection means which is image 
characteristic detection means that reads out RGB three CCD images from the 
abovementioned interpolating unit 62 in block units, and detects high-frequency 
components; and the region estimating unit 73 which applies labels proportional to the 
high-frequency components detected by the abovementioned high-frequency detection 
unit 75, and transmit transmits the labeled images to the buffer 63. the The 
abovementioned high-frequency detection unit 75 is connected bidirectionally with the 
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abovementioned control unit 22, and is controlled by the control unit 22. 
[00147] In greater detail, the abovementioned high-frequency detection unit 75 
reads out RGB three CCD images from the abovementioned interpolating unit 62 in a 
specified block size, e. g., in 8 x 8 pixel units, converts these images into frequency 
components by a universally known DCT (discrete cosine transform), determines the 
quantity of high-frequency components in each block from the frequency components 
produced by this conversion, and transmits this data to the region estimating unit 73 
in block units. 

[00148] The region estimating unit 73 applies labels proportional to the quantity 
of high-frequency components to the blocks, and transmits these labeled blocks to the 
buffer 63. 

[00 149] The correction unit 18 reads out the label information stored in the buffer 
63, subjects this label information to expansion processing in accordance with the 
proportion of down sampling performed by the abovementioned down sampling unit 61 
and the block size used in the abovementioned high-frequency detection unit 75, and 
corrects the amount of noise in pixel unit units transmitted from the noise estimating 
unit 17. 

[00150] Furthermore, in the above description, conversion into frequency 
components is accomplished by DCT. However, the present invention is of course not 
limited to this; appropriate conversion such as Fourier conversion, wavelet conversion 
or the like may be widely used. 

[00151] Next, one example of the construction of the noise estimating unit 17 will 
be described with reference to Fig. 12. 

[00152] The basic construction of the noise estimating unit 17 is similar to that of 
the noise estimating unit 17 shown in Fig. 5 in the abovementioned first embodiment; 
however, this embodiment differs from the first embodiment in that a look-up table 
unit 81 constituting look-up table means which is noise amount calculating means is 
provided instead of the abovementioned coefficient calculating unit 46, parameter ROM 
47 and function calculating unit 48. 
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[00153] The look-up table unit 81 is connected bidirectionally with the control unit 
22, and is controlled by the control unit 22; the look-up table unit 81 inputs 
information from the abovementioned average calculating unit 43, gain calculating 
unit 44 and standard value assigning unit 45, and outputs processing results to the 
correction unit 18. 

[00154] The look-up table unit 81 is a unit which constructs and records the 
relationships between the signal value level, gain, shutter speed and temperature of 
the image pickup element, and the amount of noise beforehand as a look-up table by 
means similar to those used in the abovementioned first embodiment. On the basis of 
the signal value levels of pixels of interest calculated by the abovementioned average 
calculating unit 43, the gain calculated by the abovementioned gain calculating unit 
44, information relating to the shutter speed and temperature of the image pickup 
element transmitted from the abovementioned control unit 22, and standard values 
assigned as necessary from the abovementioned standard value assigning unit 45, the 
look-up table unit 81 refers to the look-up table and estimates the amount of noise, and 
transmits this amount of noise to the correction unit 18. 

[00155] Furthermore, in the above description, the system is constructed such that 
noise reduction processing is performed at the time of shooting; however, the present 
invention is not limited to this. For example, the video signals output from the CCD 5 
may be handled as unprocessed raw data, and information from the abovementioned 
control unit 22 such as the temperature of the image pickup element during shooting, 
the gain, the shutter speed and the like may be added to this raw data as header 
information. The system may be devised such that raw data to which this header 
information has been added is output to a processing device such as a computer or the 
like, and is processed by software in this processing device. 

[00156] An example in which noise reduction processing is performed by an image 
processing program in a computer will be described with reference to Figs. 13 and 14. 
Furthermore, Figs. 13 and 14 relate to a series of processing operations. 
[00157] When the processing is started, the full color signals constituting the raw 
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data, and header information including information such as the temperature, gain, 
shutter speed and the like, are first read in (step SI). 

[00158] Next, the video signals are sampled down to a specified size (step S2), and 
an RGB three CCD image is produced by universally known linear interpolation (step 
S3). 

[00159] Color difference signals Cb and Cr are determined from the RGB thus 
produced, and colors in which these color difference signals Cb and Cr are within a 
specified range are extracted as skin color regions (step S4). 

[00160] Meanwhile, a brightness signal Y is determined from the RGB produced 
in the abovementioned step S3, an and regions in which this brightness signal Y is 
below a specified threshold value are extracted as dark regions (step S5). 
[00161] Labeling relating to the skin color regions extracted in step S4 and the 
dark regions extracted in step S5 is performed (step S6), and the applied label 
information is output (step S7). 

[00162] Next, the video signals read in in step SI are separated into respective 
color signals (step S8), and local regions of a specified size centered on the pixels of 
interest, e. g., local regions of 4 x 1 pixel units are extracted (step S9). The signal value 
levels of the pixels of interest are then calculated to be average values for the local 
regions (step S10). 

[00163] Furthermore, parameters such as the temperature, gain, shutter speed 
and the like are determined from the header information read in, in step SI; in this 
case, if required parameters are absent from the header information, specified 
standard values are assigned (step Sll). 

[00164] The amount of noise is calculated by referring to the look-up table on the 
basis of the signal value levels determined in step S10 and the parameters such as the 
temperature, gain, shutter speed and the like determined in step Sll (step S12). 
[00165] Meanwhile, the label information output in step S7 is read in, and labels 
corresponding to the pixels of interest are transmitted to step S14 (step S13). 
[00166] The amount of noise determined in step S12 is corrected on the basis of 
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the label information read in, in step S13 (step S14). 

[00167] The color signals separated in step S8 are extracted in horizontal line 
units (step S15), and universally known hysteresis smoothing is performed on the basis 
of the amount of noise corrected in step S14 (step S16). 

[00168] Then, a judgement is made as to whether or not processing has been 
completed for all horizontal lines (step S17), and in cases where this processing has not 
been completed, the processing of the abovementioned steps S9 through S16 is 
repeated. 

[00169] On the other hand, in cases where processing has been completed for all 
horizontal lines, a signal that is smoothed in the horizontal direction is output (step 
S18). 

[00170] Next, for the signal that has been smoothed in the horizontal direction, 
local regions of a specified size centered on the pixels of interest, e. g., local regions 
comprising 1x4 pixel units, are extracted (step S19), and the signal value levels of the 
pixels of interest are calculated to be average values for these local regions (step S20). 
[00171] Furthermore, parameters such as the temperature, gain, shutter speed 
and the like are determined from the header information read in, in step SI; in this 
case, if required parameters are absent from the header information, specified 
standard values are assigned (step S21). 

[00172] The amount of noise is calculated by referring to the look-up table on the 
basis of the signal value levels determined in step S20 and the parameters such as the 
temperature, gain, shutter speed and the like determined in step S21 (step S22). 
[00173] Meanwhile, the label information output in step S7 is read in, and labels 
corresponding to the pixels of interest are transmitted to step S24 (step S23). 
[00174] The amount of noise determined in step S22 is corrected on the basis of 
the label information read in in step S23 (step S24). 

[00175] The color signals smoothed in the horizontal direction in step S18 are 
extracted in vertical line units (step S25), and universally known hysteresis smoothing 
is performed on the basis of the amount of noise corrected in step S24 (step S26). 
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[00176] Then, a judgement is made as to whether or not processing has been 
completed for all of the vertical lines (step S27), and if this processing has not been 
completed, the processing of the abovementioned steps S19 through S26 is repeated. 
[00177] On the other hand, in cases where processing has been completed for all of 
the vertical lines, a signal that has been smoothed in both the horizontal direction and 
vertical direction is output (step S28). 

[00178] Afterward, a judgement is made as to whether or not processing has been 
completed for all color signals (step S29); if this processing has not been completed, the 
processing of the abovementioned steps S8 through S28 is repeated. On the other 
hand, if processing has been completed, this processing is ended. 
[00179] In this second embodiment, an effect that is more or less similar to that of 
the abovementioned first embodiment can be obtained; furthermore, since the shooting 
situation for respective regions in one shooting image is determined by performing skin 
color detection, dark area detection and high-frequency detection, and the amount of 
noise is corrected for each region, high-precision noise reduction processing which is 
suited to respective regions can be performed, so that an image with subjectively 
desirable high quality can be obtained. Furthermore, it is also possible to construct 
diverse image pickup systems according to applications by applying either the 
determination of the shooting situation of the overall signal as indicated in the first 
embodiment, or the determination of the shooting situation of respective regions as 
indicated in this second embodiment, as required. 

[00180] Furthermore, since the shooting situation for respective regions is 
estimated from signals whose size has been reduced by down sampling the video 
signals at a specified interval, high-speed processing is possible, and the memory size 
used for operation can be reduced; accordingly, an image pickup system can be 
constructed at a low cost. 

[00181] Furthermore, since a look-up table is used when the amount of noise is 
calculated, high-speed processing can be performed. 

[00182] Moreover, the present invention is not limited to the embodiment 
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described above; various modifications and applications are possible within a range 
that involves no departure from the spirit of the invention. 

[00183] In this invention, it is apparent that working modes difforont in differing 
over a wide range can be formed on the basis of this invention without departing from 
the spirit and scope of the invention. This invention is not restricted by any specific 
embodiment except boing as may be limited by the appended claims. 
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